Inservice trainings for Shiraz University of Medical Sciences employees: Effectiveness assessment by using the CIPP model.

INTRODUCTION
Nowadays, the employees` in-service training has become one of the core components in survival and success of any organization. Unfortunately, despite the importance of training evaluation, a small portion of resources are allocated to this matter. Among many evaluation models, the CIPP model or Context, Input, Process, Product model is a very useful approach to educational evaluation. So far, the evaluation of the training courses mostly provided information for learners but this investigation aims at evaluating the effectiveness of the experts' training programs in SUMS and identifying its pros and cons based on the 4 stages of the CIPP model.


METHOD
In this descriptive analytical study, done in 2013, 250 employees of SUMS participated in in-service training courses were randomly selected. The evaluated variables were designed using CIPP model and a researcher-made questionnaire was used for data collection; the questionnaire was validated using expert opinion and its reliability was confirmed by Cronbach's alpha (0.89). Quantitative data were analyzed using SPSS 14 and statistical tests was done as needed.


RESULTS
In the context phase, the mean score was highest in solving work problems (4.07±0.88) and lowest in focusing on learners' learning style training courses (2.68±0.91). There is a statistically significant difference between the employees` education level and the product phase evaluation (p<0.001).  The necessary effectiveness was not statistically significant in context and input level (p>0.001), in contrast with the process and product phase which showed a significant deference (p<0.001).


CONCLUSION
Considering our results, although the in-service trainings given to sums employees has been effective in many ways, it has some weaknesses as well. Therefore improving these weaknesses and reinforcing strong points within the identified fields in this study should be taken into account by decision makers and administrators.


Introduction
N owadays, the employees` inservice training has become one of the core component in survival and success of any organization. In the past decade, the rapid evolution of science and information technology, not only has heighten the need for but also mandate such trainings. Living in the age of information and technology advancement, the employees can no longer fulfill their diverse duties by relying on their old and out dated knowledge and skills. Moreover, the organizations also cannot remain indifferent to these advances, since they must compete with parallel organizations which utilize these new technologies. These facts has pointed out a very important need for organizational survival and improvement which is essential designing, implementing and constant evaluating of inservice educations. In addition, by extending occupational knowledge, attitude, and teaching group work through cooperative learning, the inservice training can enhance human resources capability and help them to do their job more efficiently (1). It also will result in lifelong learning since philosophically speaking, inservice education has origins in it (2).
Unfortunately, despite the importance of training evaluation, a small portion of considerable resources are actually allocated to this matter. What's more, many expensive in-service training programs which are even designed thoughtfully lack evaluation (3).
Homen believed that whatever is done to gather information, improve training problems and provide feedbacks for managers can be considered as training evaluation. Therefore, in order to make intelligent decisions and as a result improve the quality of training programs an evaluation based on these factors must be done. Such a feedback, if designed properly, will guarantee the success of training programs (4).
As Stuffelbeam has proven in 1983, among many evaluation models, the CIPP model or Context, Input, Process, Product model is a very useful approach to educational evaluation which as a standard, provides a systematic and constructional way of evaluation in many training programs. Based on CIPP model the most important goal of evaluation is the training courses improvement rather than its confirmation. In addition, instead of focusing only on an individual improvement, this model offers information which can be used by decision makers in a training institute for ongoing evaluation of the programs. By providing an organized feedback of the current affairs, this model helps the managers to prioritize the essential needs and allocate available resources to more effectual activities (5,6).
In context evaluation, primary programming and the program environment are examined. Since in this phase, the program may have not been started yet, the need to implement the program and required facilities may also be examined (7).
The input evaluation phase which starts after the execution approval includes: program time table, facilities and human resources management (7). The goal of input evaluation is facilitating the implementation of the program designed in the context phase. In addition this phase also focuses on human and financial facilities, politics, training changes, instructional strategies, obstacles and limitations of the training system (8).
In the process evaluation phase, the program itself goes under close observation and the changes of input during the program and the later on output is investigated. In fact, it is a quality control phase during its execution (7). The performance of process evaluation is largely depended on the implementation of the two previously mentioned phases; in other words, if the context and input evaluations are done properly, the implementation of the program is defiantly easier and more successful; as a result the process phase face less obstacles (8).
The final phase or product evaluation is considered as the goal of evaluation. By demonstrating all the desired and undesired outcomes, this phase shows what has or has not been done (7). As Stuffelbeam put it, product evaluation is measuring and judging the achievement of predefined goals. In product evaluation this measurement and interpretation is done not only at the end of the program but also during the program execution and completing each task. This is due to the periodic feature of the evaluation process, in other words evaluation is a process that should be done constantly and the results of each phase should be utilized to improve the previous phase and program the next phase. In some cases product evaluation is expanded until it covers long term effects of a program and its positive and negative products (8).
So far, the evaluation of the training courses mostly provided information for learners but this investigation aims at evaluating the effectiveness of the experts' training programs in Shiraz University of Medical Sciences based on the CIPP model, and identifying weak and strong points of the program, and providing results for administrators and decision makers, and can obtain guidelines for designing better quality programs.

Methods
In 2013, this analytical and illustrative study was done on all the employees of Shiraz University of Medical Sciences (2219) who took part in different inservice trainings. We used Kerjesy and Morgan formula for calculating the sample size.
In an attempt to increase the accuracy of the study and compensate for possible low response rate, 10 percent was added to the actual sample size. After gaining informed consent, 250 employees which were selected randomly entered the survey. A questionnaire based on Stuffelbeam' s CIPP evaluation model was designed (9). This questionnaire consists of two parts: The first part collect the demographic data such as: gender, education level, age, employment type, work experience, field of study, type of job and place of work ;whereas the second part which contained 62 (58 quantitative and 4 qualitative) questions gathered the effectiveness data. The employees ranked each quantitative item based on a five rating scale which was: (1=very low, 2=low, 3=average, 4=high, 5=very high). Achieving 3 points was considered the cut point of effectiveness for each phase which meant that the effectiveness is undesirable if the average of points given to the items in each phase is less than 3 (1-2.99) and desirable if it is 3 or more (3)(4)(5), providing that the (p<0.001) in each case.
The qualitative questions were open -ended and used for gathering personal opinions and suggestions. The questionnaire`s questions were based on four phases of CIPP model and previous studies,. These phases are: 1) Context Phase: Training infrastructure, opportunities, satisfaction, training needs assessment, training resources, learning styles, location status, and manager support.
2) Input Phase: Staff, training content, program designing, budget and financial resources, training environment, supports staff, and instructors.
3) Process: Training process, learning, number of instructors, instructors' performance, support and encouragement, performance of the library and announcement. 4) Product or outcomes Phase: The quality of educational content, the rate of reaching goals, effectiveness of trainings, enhancing the knowledge and skills, the motivation for getting educational certificate, evaluation of trainees' information and empowering the staff.
Questions 1 to 10 evaluate the context component, Questions 11 to 25 evaluate the input component, Questions 26 to 48 evaluates the process component, Questions 49 to 58 evaluates the product component, and Questions 59,60,61,62 measures the results, qualitatively. The reliability of the questionnaire was determined by experts through Cronbach's alpha test. The cronbach's alpha coefficient was 0.89 which suggests that the questionnaire has the required reliability. The validity was also established using experts opinion. The questionnaires were distributed and collected via face to face or email approach. The data were then analyzed using descriptive and inferential statistics including distribution of population, mean, standard deviation, correlation coefficient, and t-test.

Results
In the present study, 68% of the participants were women and 32% were men. 163 participants (67.2%) had Bachelor's, 57 (23.4%) had associate and 23(9.4%) had Master's and Ph.D degrees. The mean and standard deviation of the study participants' age and work experience were (36.3±6.14) and (11.25±6.79), respectively. The employees enrolled in the study were from deferent departments such as healthcare, administrative -financial, educational-cultural and technicalengineering departments. The gender affected the process phase `s results significantly (p=0.045); in contrast with the other phases in which the gender didn't play any role (p>0.001). In the product phase, the educational degree of the employees which included Associate, Bachelor's, Master's and Ph.D degrees significantly affected the results (p<0.001).
Quantitative data showed that in the context phase, the mean score was highest in solving work problems (4.07±0.88) and lowest in focusing on learners' learning style training courses (2.6±0.94). Although the total score achieved in the context phase was 3.17 overall, this phase didn't have the necessary effectiveness (p>0.001) ( Table 1).
In the input phase, the maximum mean score was related to holding courses in office hours and the instructors' interest in teaching (3.6±0.731) and the minimum mean score was related to needs assessment of training courses (2.4±1.63). The total score was reported to be 3.03 Overall, but like the context phase, this phase also lacked the necessary effectiveness (p>0.001) ( Table 2).
In the process phase of the evaluation, using new educational methods and technologies had the maximum (3.50±0.862) and sufficient budget and facilities for educational needs had the minimum mean score (2.6±0.855). The total score was reported to be 3.15. Overall and it was statistically significant (p<0.001), therefore it can be concluded that this phase had the necessary effectiveness (Table 3).
In the product phase, "the quality of materials offered" had the highest mean score (3.2±0.749) while the lowest mean score belonged to"the offering feedback item" (2.8±0.966). The total score was reported to be 3.13, Overall, and since it was statistically significant, the product phase has the necessary effectiveness (p<0.001) ( Table 4).
The CIPP Model full reports based on the participants answer are as follows: Regarding the question "how to hold a better training course", the most effective factors included: Appropriate place selection with respect to proximity to workplace, holding virtual and distance courses, motivating learners and providing appropriate situation for attending training courses, in the context phase. Moreover, utilizing instructors from inside and outside of the organization, needs assessment, holding courses based on the employees work requirements and degrees, programming before implementation, holding the courses in office hours, Allocating enough budget, educational resources and facilities and general environment of the courses were the most important factors mentioned in the input phase, and last but not least regarding the process phase, the main contributing factors were: holding practical courses, applying new educational methods such as learner-based teaching, reducing the number of learners, allocating enough time to each course and announcing the dates before head.
In response to the question" whether evaluation can result in the improvement of programs and why" in the product phase: The most influential factors in inservice training evaluation included: 1. Methodical evaluation 2. Weaknesses and strong points identification 3. Attending these training programs should have a better motivation rather than just receiving a certificate in education 4. Improving the quality of education and level of knowledge and science 5. Improving programs 6. Offering feedbacks

Discussion
The results of the current study have shown that the effectiveness of inservice training depends on various factors. Therefore, in order to be effective, these factors must improve.
Regarding the training courses presented to SUMS` employees, the CIPP evaluation done in this survey indicated that, there are some strong points and some weaknesses. According to our study, high interest in taking these courses and the training which result in solving work problems are the program strong points while the weaknesses included: the lack of educational facilities for holding virtual  (10)(11)(12). Like Hosseini et al in 2008, our study have found that factors such as allocating budget, financial and educational resources, needs assessment of training courses and take the staffs` problems in to consideration while choosing the subject and content of these courses, to be the most important issues in the component of input (1).
The results of the present study showed that the most influential factors which decrease the effectiveness of the programs are ignoring needs assessment procedures, educational-welfare conditions and facilities (12). Also, our results are in agreement with Mahmoodi et al. (2012) and Yarmohammadian et al.'s. (2011) studies in which needs assessment are mentioned as the greatest change of innovation in sustaining the inservice training programs which is lower than expected (2,13). Decision makers and education programmers should pay more attention to the necessity of needs assessment that is the first and most important stage in each organization (13).
One of other effective factors is to utilize instructors from inside and outside of the organization. In this way, the program can benefit from the insiders awareness due to their familiarity with the organization problems and also the specialty and experience of outsiders in special fields. Other factors include: programming before implementation and holding courses compatible with the employees' work and degree during their work hours which are in agreement with Mahmoodi et al.'s study in 2010 (13).
Other training effectiveness obstacles included: lack of budget and facilities for educational needs, insufficient educational contents and time allocated to them. Items such as goals introduction methods, effective teaching style, quality of educational tools, learners' motivation enhancement and the use of practical examples during teaching and relatively up-to-date contents must also be taken in to consideration and revise. Khotami's study (2012) who also conducted an evaluation based on CIPP model in Saudi Arabia also emphasized on the importance of facilities and equipment, training environment and capability of instructors as a useful controlling model (14).
In the product phase, the employees believed that one of the main weaknesses of the evaluation is the fact that many attend these courses with sole purpose of receiving educational certificate. Other weak points in this phase included the lack of utilization of program results by administrators and their ignorance of such programs' benefits, as well as lack of feedbacks to staffs. Likewise, in Hosseinpour et al's study (2012) which also found motivations such as a guaranteed certificate for work promotion and raise upon fulfillment of the course as one of the trainings weak points, since it could consequently lead to lack of active participation in training and less effective program (16).
Also, results from Akhlaghi and Yarmahmmodian (2011) and Heidari et al. 's. (2005) studies showed that evaluation of the product phase was not desirable and even lower than expected indicating some problems; therefore a sustainable evaluation and identification of weak and strong points of programs should be used to improve the quality level of training courses (17,18).

Conclusion
In conclusion, although the inservice trainings given to sums employees have been effective in many ways, they also have some weaknesses especially in the input and context phases, in which more effectiveness was expected. The product and process phases of the training, however, were desirably effective. Therefore, Improving these weaknesses and reinforcing strong points within the identified fields in this study should be taken into account by decision makers and administrators. In this regard it seems that to enhance the positive points of these training courses, an environment of creativity and productivity in all phases must be created and actively supported by the administration and those in charge of inservice training program. If done correctly and continuously, this may also lead to weak points` improvement. Moreover, needs assessment, electronic learning facilitation and the employees points of view regarding the time and place and content of the trainings must be taken under consideration in.